Adding OpticalFlow Algorithms
 | This page page is under construction. |
Introduction
The optical flow algorithms dictate how motion vectors are estimated between consecutive frames. In general, each algorithm can be understood as a method that takes a sequence of image frames and produces a set of motion vectors representing the apparent displacement of pixels or features over time. The behavior and accuracy of each algorithm can be adjusted through a set of algorithm-specific parameters.
The steps to implement a new optical flow algorithm are:
- Define the new optical flow algorithm.
- Define the algorithm parameters and their valid ranges.
Next, we will showcase how to implement any optical flow algorithm following the previous steps.
Define the Optical Flow Algorithm
The RidgeRun Video Stabilization Library is extensible and can adopt different optical flow algorithms. Apart from the usual constructor method, the IOpticalFlow interface describes how to implement new algorithms with the following methods:
- The GetRotation method that applies the optical flow motion detection and outputs the estimated rotation data.
- The GetHomography method that estimates the frame-to-frame homography from optical flow.
- The SetRuntimeSettings method that applies runtime settings to the optical flow instance.
- The GetRuntimeSettings method that retrieves the current runtime settings.
- The SetCameraMatrices method that configures the camera intrinsic and distortion parameters.
Additionally, the algorithm can be included in the static builder method of the interface.
Add the Constructor Method
This method should receive a shared pointer to the corresponding optical flow parameter instance and, optionally, a shared pointer to the runtime settings instance. Since the parameter object is passed through the base interface type, it must be dynamically cast to validate that it corresponds to the expected algorithm.
ExampleOpticalFlow::ExampleOpticalFlow(
const std::shared_ptr<OpticalFlowParams> params,
const std::shared_ptr<IRuntimeSettings> settings) {
std::shared_ptr<ExampleOpticalFlowParams> casted =
std::dynamic_pointer_cast<ExampleOpticalFlowParams>(params);
if (casted == nullptr) {
throw RuntimeError{
RuntimeError::IncompatibleParameters,
"The optical flow parameters are incompatible. "
"Use ExampleOpticalFlowParams"};
}
params_ = casted;
if (settings != nullptr) {
SetRuntimeSettings(settings);
}
}
The constructor is responsible for validating the optical flow parameter type and initializing the runtime settings when provided.
Extend Static Builder
To enable the IOpticalFlow interface to instantiate custom algorithms, it is required to:
- Extend the OpticalFlowAlgorithms enumerator class.
- Add the corresponding case to the switch in the builder method.
Extend OpticalFlowAlgorithms enum
enum class OpticalFlowAlgorithms {
kInvalid = -1,
kDISOpenCV = 0,
kPyrLKOpenCV = 1,
kPyrLKCUDA = 2,
/* add new algorithm */
kExampleOpticalFlow,
};
Modify Builder Method
std::shared_ptr<IOpticalFlow> IOpticalFlow::Build(
const OpticalFlowAlgorithms impl,
const std::shared_ptr<IUndistort> undistort,
const std::shared_ptr<OpticalFlowParams> params,
const std::shared_ptr<IRuntimeSettings> settings) {
static const std::string err_message =
"The requested implementation is missing or has not been compiled: ";
switch (impl) {
#ifdef HAVE_OPENCV
case OpticalFlowAlgorithms::kDISOpenCV:
return std::make_shared<DisOpticalFlowOpenCV>(undistort, params,
settings);
case OpticalFlowAlgorithms::kPyrLKOpenCV:
return std::make_shared<PyrLKOpticalFlowOpenCV>(params, settings);
case OpticalFlowAlgorithms::kPyrLKCUDA:
return std::make_shared<PyrLKCUDA>(params, settings);
#endif
case OpticalFlowAlgorithms::kExampleOpticalFlow:
return std::make_shared<ExampleOpticalFlow>(params, settings);
default:
throw RuntimeError{RuntimeError::NotImplemented,
err_message + kImplementationNames.at(impl)};
}
return nullptr;
}
Define the GetRotation Method
The GetRotation method applies the optical flow algorithm to the previous and current frame and stores the estimated rotation data in the output sensor payload.
RuntimeError ExampleOpticalFlow::GetRotation(
std::shared_ptr<SensorPayload> output_payload,
const std::shared_ptr<IImage> frame) {
/* Implement optical flow rotation estimation here */
return RuntimeError{RuntimeError::Ok};
}
Define the GetHomography Method
The GetHomography method estimates the frame-to-frame homography using the previous frame state and the current input frame.
RuntimeError ExampleOpticalFlow::GetHomography(
std::array<float, 9> &output_homography,
const std::shared_ptr<IImage> frame) {
/* Implement optical flow homography estimation here */
return RuntimeError{RuntimeError::Ok};
}
Define the SetRuntimeSettings Method
The SetRuntimeSettings method updates the runtime settings used by the optical flow instance. Since runtime settings are passed through the base interface type, they must be dynamically cast to validate that they match the expected implementation.
RuntimeError ExampleOpticalFlow::SetRuntimeSettings(
const std::shared_ptr<IRuntimeSettings> settings) {
auto settings_cast =
std::dynamic_pointer_cast<ExampleRuntimeSettings>(settings);
if (nullptr == settings_cast && nullptr != settings) {
return RuntimeError{
RuntimeError::IncompatibleParameters,
"The runtime settings are incompatible. Use ExampleRuntimeSettings"};
}
this->settings_ = settings_cast;
return RuntimeError{};
}
Define the GetRuntimeSettings Method
The GetRuntimeSettings method returns a copy of the current runtime settings used by the optical flow instance.
std::shared_ptr<IRuntimeSettings> ExampleOpticalFlow::GetRuntimeSettings() {
auto new_settings = std::make_shared<ExampleRuntimeSettings>();
if (nullptr != this->settings_) {
*new_settings = *this->settings_;
}
return new_settings;
}
Define the SetCameraMatrices Method
The SetCameraMatrices method configures the camera intrinsic and distortion parameters used by the algorithm. Algorithms that require camera calibration data should store and validate these parameters. If camera matrices are not applicable to a particular implementation, the method may return a NotImplemented error.
RuntimeError ExampleOpticalFlow::SetCameraMatrices(
const std::array<float, 9> &matrix,
const std::vector<float> &dist,
const int cal_width,
const int cal_height) {
(void)matrix;
(void)dist;
(void)cal_width;
(void)cal_height;
return RuntimeError{
RuntimeError::NotImplemented,
"ExampleOpticalFlow::SetCameraMatrices is not implemented"};
}
